1. Field of the Invention
The present invention relates generally to methods, systems, and program products for calculating, detecting, observing, or validating operating characteristics, conditions, and metrics, especially quality of service metrics, of a system, and using the quality of service metrics to manage a service.
2. Background Art
Various business considerations have led a growing number of organizations to rely on external vendors to support their needs, such as order entry, order fulfillment, billing, lock box services, payment processing, customer service, help desk and technical support, and warranty service. Much of the day-to-day data on vendor performance from vendors are not available to the vendee, and typically the vendee organization ends up with its own systems and acceptance tests to validate the vendor services. The readiness of the vendor-services has been evaluated based primarily on the actual test execution results, including customer feedback. New metrics were derived to measure the degree of risk associated with a variety of test case failures such as time constraints and functionalities not enabled, bad fixes, and defects not fixed during successive iterations. The relationship of these metrics to the actual cause was validated through explicit communications with the vendor and the subsequent actions to improve the quality and completeness of the delivered service.
Appropriate use of metrics is vital to any project. These metrics help track aspects of an ongoing outsourced, customer support project, such as changing requirements, rates of finding and fixing defects, and growth in size and complexity of code. From the testing point of view, the metrics typically focus on the quantity and quality of service, the progress of testing, and the effectiveness of testing. Examples of typical metrics include product or release size over time, defect discovery rate over time, defect backlog over time, test progress over time (plan, attempted, actual), and percentage of test cases attempted.
The trend toward increased outsourcing of customer service and support demands the regular use of new metrics and methodology to assure adequate quality and schedule integrity. One of the challenges a vendee organization faces is the evaluation of the outsourced customer service and support in terms of functionality, performance, etc. Because of the implicit business risks in outsourcing customer services and support, the contractual commitments for quality and completeness are generally at a high level, and typically the vendee organization defines and executes its own acceptance test to validate expectations.
Meaningful metrics help users and developers evaluate performance and performance details and answer questions. These meaningful metrics can be derived from the execution data and used in an operational environment
Formulating meaningful metrics requires conducting empirical studies—either experiments or case studies. Experiments give users and vendors more control, but they are harder to conduct and more expensive than a case study. Moreover, case studies usually yield more data that is accessible.
Most metric calculation engines only provide a high level framework to process the required calculations, and require user written code, frequently extensive user written code, to identify and capture inputs and outputs characterizing the metrics and to then structure the metric engine to perform necessary metric calculations. This code may be complex, and require training and experience to write.
Thus a need exists for an integrated system, independent of and not dependent on user written code to identify and capture inputs and outputs which characterize the process and its metrics and to then structure the metric engine to perform the requested and desired metric calculations. Such a system would respond to a simple, user submitted metric calculation request, e.g., performing business logic against incoming transaction data to translate the submitted transaction data into the required metrics, and present the results to the user.